Optimal Correction for Guessing in Multiple-Choice Tests

Building on Item Response Theory we introduce students’ optimal behavior in multiple-choice tests. Our simulations indicate that the optimal penalty is relatively high, because although correction for guessing discriminates against risk-averse subjects, this effect is small compared with the measurement error that the penalty prevents. This result obtains when knowledge is binary or partial, under different normalizations of the score, when risk aversion is related to knowledge and when there is a pass-fail break point. We also find that the mean degree of difficulty should be close to the mean level of knowledge and that the variance of difficulty should be high.

Are multiple-choice tests a suitable method to evaluate students' knowledge? : against the extensive use of multiple-choice evaluation test. '¿Son los test de opción múltiple un método adecuado para evaluar el conocimiento de los estudiantes? : en contra del uso extensivo del test de evaluación de opción múltiple'.

Resumen basado en el de la publicación

Large-scale assessment of language proficiency: theoretical and pedagogical reflections on the use of multiple-choice tests

The new requirement placed on students in tertiary settings in Spain to demonstrate a B1 or a B2 proficiency level of English, in accordance with the Common European Framework of Reference for Languages (CEFRL), has led most Spanish universities to develop a program of certification or accreditation of the required level. The first part of this paper aims to provide a rationale for the type of test that has been developed at the Universidad Politécnica de Madrid for the accreditation of a B2 level, a multiple choice version, and to describe how it was constructed and validated. Then, in the second part of the paper, the results from its application to 924 students enrolled in different degree courses at a variety of schools and faculties at the university are analyzed based on a final test version item analysis. To conclude, some theoretical as well as practical conclusions about testing grammar that affect the teaching and learning process are drawn. RESUMEN. Las nuevas exigencias sobre niveles de competencia B1 y B2 en inglés según el Marco Común Europeo de Referencia para las Lenguas (MCERL) que se imponen sobre los estudiantes de grado y posgrado han llevado a la mayoría de las universidades españolas a desarrollar programas de acreditación o de certificación de estos niveles. La primera parte de este trabajo trata sobre las razones que fundamentan la elección de un tipo concreto de examen para la acreditación del nivel B2 de lengua inglesa en la Universidad Politécnica de Madrid. Se trata de un test de opción múltiple y en esta parte del trabajo se describe cómo fue diseñado y validado. En la segunda parte, se analizan los resultados de la aplicación del test a gran escala a un total de 924 estudiantes matriculados en varias escuelas y Facultades de la Universidad. Para terminar, se apuntan una serie de conclusiones teóricas y prácticas sobre la evaluación de la gramática y de qué modo influye en los procesos de enseñanza y aprendizaje.

On the strategic equivalence of multiple-choice test scoring rules

A disadvantage of multiple-choice tests is that students have incentives to guess. To discourage guessing, it is common to use scoring rules that either penalize wrong answers or reward omissions. These scoring rules are considered equivalent in psychometrics, although experimental evidence has not always been consistent with this claim. We model students' decisions and show, first, that equivalence holds only under risk neutrality and, second, that the two rules can be modified so that they become equivalent even under risk aversion. This paper presents the results of a field experiment in which we analyze the decisions of subjects taking multiple-choice exams. The evidence suggests that differences between scoring rules are due to risk aversion as theory predicts. We also find that the number of omitted items depends on the scoring rule, knowledge, gender and other covariates.

Using constructed-response and multiple-choice questions in undergraduate examinations

This paper looks at the use of constructed-response and multiple-choice questions in first year management examinations involving 2364 students over a three semester period in 2008-9. It also compares student’s performance in research and analytical assignments with their results in Multiple-Choice tests. The results show that students who perform poorly in non-multiple-choice tests are slightly advantaged by multiple-choice tests. However, students who display high distinction scores in other pieces of assessment do not receive any advantage from multiple-choice testing. Interestingly, the research also highlighted that female students perform better in their articulation of management techniques overall, but do not gain the same comparative advantage from multiple choice testing. The analysis of the assessment methods was extremely useful in predicting the level of failures for the unit in the subsequent year. Finding the right balance of assessment tasks in first year university business courses is extremely difficult, using multiple-choice tests can be useful tool as part of the assessment mix in management units.

The effect of the introduction of teaching interventions and assessment strategies in first year management units

This Scholarship of Teaching research aimed to ultimately become a report and a series of recommendations on the teaching of first year Management units in universities. Many of the Australian universities benchmark themselves against the leaders in the Course Experience Questionnaires (CEQ) (Ramsden 1991) of graduating students in Australia. Most Australian universities have their own student satisfaction evaluation method. In this study the subject university uses the Student Evaluation of Teachers and Units (SETU) as their internal method of measuring the effectiveness of teaching on units on a cohort by cohort basis. This project, using one of the largest units in a Business and Law Faculty had specific measurable objectives; the results of the unit teams' interventions demonstrate that it is possible to increase student satisfaction during their transition year by as much as 11.5%. The ten recommendations ranging from changes to unit objectives, assessment methods and updated methods of marking, communication and student support, if acted upon, should have ongoing benefits for teaching and learning of first year management units.

An Australian study comparing the use of multiple-choice questionnaires with assignments as interim, summative law school assessment

o the author’s knowledge, this is the first Australian study to empirically compare the use of a multiple-choice questionnaire (MCQ) with the use of a written assignment for interim, summative law school assessment. This study also surveyed the same student sample as to what types of assessments are preferred and why. In total, 182 undergraduate property law students participated in this study. Results showed that scores for the MCQ (assessing five topics) and assignment (assessing one topic) followed a similar distribution. This indicates that an MCQ does not necessarily skew students towards higher grades than an assignment. Results also showed significant but low correlations of test scores across instruments. When asked which instrument best assessed their knowledge of property law, students expressed a strong preference for an assignment over an MCQ or examination. Comments revealed a strong belief that, because lawyers write, law schools must assess legal writing – a skill not captured by MCQs. This study is important as many Australian law schools face increasing marking loads due to higher student numbers and compulsory mid-term assessments. This article endorses the use of MCQs but only as part of a diverse suite of law school assessment.

Sustainable assessment for large science classes : non-multiple choice, randomised assignments through a Learning Management System

This paper reports on the development of a tool that generates randomised, non-multiple choice assessment within the BlackBoard Learning Management System interface. An accepted weakness of multiple-choice assessment is that it cannot elicit learning outcomes from upper levels of Biggs’ SOLO taxonomy. However, written assessment items require extensive resources for marking, and are susceptible to copying as well as marking inconsistencies for large classes. This project developed an assessment tool which is valid, reliable and sustainable and that addresses the issues identified above. The tool provides each student with an assignment assessing the same learning outcomes, but containing different questions, with responses in the form of words or numbers. Practice questions are available, enabling students to obtain feedback on their approach before submitting their assignment. Thus, the tool incorporates automatic marking (essential for large classes), randomised tasks to each student (reducing copying), the capacity to give credit for working (feedback on the application of theory), and the capacity to target higher order learning outcomes by requiring students to derive their answers rather than choosing them. Results and feedback from students are presented, along with technical implementation details.

Short answer versus multiple choice examination questions for first year chemistry

Multiple choice (MC) examinations are frequently used for the summative assessment of large classes because of their ease of marking and their perceived objectivity. However, traditional MC formats usually lead to a surface approach to learning, and do not allow students to demonstrate the depth of their knowledge or understanding. For these reasons, we have trialled the incorporation of short answer (SA) questions into the final examination of two first year chemistry units, alongside MC questions. Students’ overall marks were expected to improve, because they were able to obtain partial marks for the SA questions. Although large differences in some individual students’ performance in the two sections of their examinations were observed, most students received a similar percentage mark for their MC as for their SA sections and the overall mean scores were unchanged. In-depth analysis of all responses to a specific question, which was used previously as a MC question and in a subsequent semester in SA format, indicates that the SA format can have weaknesses due to marking inconsistencies that are absent for MC questions. However, inclusion of SA questions improved student scores on the MC section in one examination, indicating that their inclusion may lead to different study habits and deeper learning. We conclude that questions asked in SA format must be carefully chosen in order to optimise the use of marking resources, both financial and human, and questions asked in MC format should be very carefully checked by people trained in writing MC questions. These results, in conjunction with an analysis of the different examination formats used in first year chemistry units, have shaped a recommendation on how to reliably and cost-effectively assess first year chemistry, while encouraging higher order learning outcomes.

Antimicrobic susceptibility and plasmid profile analysis as identity tests for multiple blood isolates of coagulase-negative staphylococci

We compared a disk diffusion antimicrobic susceptibility panel with plasmid DNA profiles as tests for identity of 106 isolates of coagulase-negative staphylococci cultured from the blood of 45 patients on multiple occasions. The antimicrobic panel included penicillin, oxacillin, clindamycin, trimethoprim-sulfamethoxazole, chloramphenicol, tetracycline, tobramycin, kanamycin, and gentamicin. Nineteen patterns of antimicrobic susceptibility were found. The most common pattern was present in 25% of the isolates, and at least one isolate from 31% of the patients had this pattern. Forty-seven distinct plasmid DNA profiles were found. The most common plasmid profile was present in 8.5% of the isolates, and at least one isolate from 15% of the patients had this profile. Twenty-eight patients had multiple isolates that were identical by plasmid profile analysis. Twenty-seven (96%) of these patients had isolates that were also identical by antimicrobic susceptibility. Nineteen patients had multiple isolates that were different by plasmid profile analysis. In 18 (95%) of these patients, the isolates were also different by antimicrobic susceptibility. Although plasmid DNA profile analysis is a more discriminating tool, these data confirm that a selected disk diffusion antimicrobic susceptibility panel may be used to screen multiple blood isolates of coagulase-negative staphylococci for identity or differences.

English tests : graded multiple choice : A1. 'Test de Inglés : respuesta de múltiple elección : A1'.

Se trata de una serie de libros de test de respuesta de múltiple elección sobre el idioma inglés. La serie abarca los seis niveles del Marco Común Europeo, de A1 a C2, con un libro dedicado a cada nivel. Cada libro se compone de doce test de 30 minutos, cada uno de los cuales puede ser fácilmente realizado y corregido. Pero no son sólo pruebas para determinar los conocimientos de un estudiante en gramática y vocabulario, también están diseñados para ser utilizados como herramientas de enseñanza valiosas por derecho propio, permitiendo a profesores y estudiantes revisar las estructuras más comunes y las dificultades del lenguaje inglés, así como controlar su progreso a medida que avanzan.

English tests : graded multiple choice : A2. 'Test de Inglés : respuesta de múltiple elección : A2'.

Se trata de una serie de libros de test de respuesta de múltiple elección sobre el idioma inglés. La serie abarca los seis niveles del Marco Común Europeo, de A1 a C2, con un libro dedicado a cada nivel. Cada libro se compone de doce test de 30 minutos, cada uno de los cuales puede ser fácilmente realizado y corregido. Pero no son sólo pruebas para determinar los conocimientos de un estudiante en gramática y vocabulario, también están diseñados para ser utilizados como herramientas de enseñanza valiosas por derecho propio, permitiendo a profesores y estudiantes revisar las estructuras más comunes y las dificultades del lenguaje inglés, así como controlar su progreso a medida que avanzan.

English tests : graded multiple choice : B1. 'Test de Inglés : respuesta de múltiple elección : B1'.

Se trata de una serie de libros de test de respuesta de múltiple elección sobre el idioma inglés. La serie abarca los seis niveles del Marco Común Europeo, de A1 a C2, con un libro dedicado a cada nivel. Cada libro se compone de doce test de 30 minutos, cada uno de los cuales puede ser fácilmente realizado y corregido. Pero no son sólo pruebas para determinar los conocimientos de un estudiante en gramática y vocabulario, también están diseñados para ser utilizados como herramientas de enseñanza valiosas por derecho propio, permitiendo a profesores y estudiantes revisar las estructuras más comunes y las dificultades del lenguaje inglés, así como controlar su progreso a medida que avanzan.